Skip to content

Name the database row and Modal call in v2 telemetry, and emit the Simulation Entrypoint's log fields - #1092

Merged
anth-volk merged 14 commits into
mainfrom
v2plan/c1-34-v2-telemetry-record-ids
Nov 18, 2026
Merged

anth-volk merged 14 commits into
mainfrom
v2plan/c1-34-v2-telemetry-record-ids

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1091

Each v2 Modal function's telemetry operation now names the API v1 row its call works on and its own Modal call. An operation that Modal interrupted in production then leads straight to its row.

New operation attributes

Function database_table database_row_id from modal_call_id
run_single_simulation_us, run_single_simulation_uk simulation_runs context["simulation_run_id"] the worker's call
run_single_simulation_segment_us simulation_runs new keyword simulation_run_id, which the US worker passes to each segment it starts the segment's call
coordinate_report (budget-window years included) report_runs context["report_run"]["report_run_id"] the coordinator's call
calculate_household_us, calculate_household_uk with household_job_id household_jobs household_job_id the function's call
the same with household_report_run_id report_runs (a household report run is a report_runs row) household_report_run_id the function's call

A call that works on no detailed-save row (a comparison-path call, a standalone simulation, a release validation's household calculation) names only modal_call_id. Only a well-formed UUID is named: a malformed ID is left out, and the function refuses it as before.

Why these names. The observability package (policyengine-observability 3.0.7) drops every attribute whose key contains household, reform, payload or person, among others, so household_job_id would never reach a log. job_id is reserved for one annual v1 Modal invocation, and the Simulation Entrypoint already logs record/record_id with other values. One table-and-row pair works for every function, and the observability guide's identifier table now lists all three names.

Off metrics. The IDs go to the operation's log records and span. Operation metrics carry only the service, the operation name and kind, and the outcome, under the package's metric allow list, which names none of these keys. A test reads the filtered metric attributes to confirm this.

The Simulation Entrypoint's log fields

The observability package's log handler emits only a log record's policyengine_attributes; any other key passed in extra= stays on the Python LogRecord. 31 of the Entrypoint's 34 log calls that pass fields passed them only that way, so their records in Cloud Logging carried the event name and no fields: no household job, report, run or error type. A key such as household_job_id would also have been dropped by the key filter above.

  • A new log_extra(**fields) in policyengine_simulation_observability.log_records puts each field in both places, and every Entrypoint log call that passes fields uses it, including household_job_outcome_unknown, which Answer 500 when a household job's commit is unconfirmed, and move sim's database code to the SQLAlchemy ORM #1090 added. A field named like a LogRecord attribute goes only to policyengine_attributes, since logging raises for it.
  • Rows are named with the same database_table and database_row_id, through database_row_attributes(table, row_id) and one DatabaseTable type shared with the v2 Modal functions:
Log events Old fields database_table
household_call_unconfirmed, household_job_not_failed, household_job_not_read, household_call_not_recorded household_job_id household_jobs
household_report_call_unconfirmed, household_report_call_not_recorded, household_report_run_not_failed report_id reports
economy_call_unconfirmed, economy_run_not_failed, economy_call_not_recorded record, record_id reports or simulations
budget_window_not_started, budget_window_call_unconfirmed record, record_id reports
budget_window_year_deferred record_id (a report run) report_runs
  • budget_window_call_unconfirmed's started_years is joined into one string, since the runtime drops lists.
  • The Entrypoint sets its package logger to INFO. The root logger stays at WARNING, which discarded its INFO records (budget_window_year_deferred, invalid_simulation_entry_bearer_token) before the handler saw them.
  • New rule sim.observability.log-fields-through-log-extra, enforced by tests/test_log_fields.py, which checks that every Entrypoint log call with extra= uses log_extra and that no field name written as a keyword contains a word the package drops. It has proof fixtures, and docs/generated/rules.md and AGENTS.md are rendered again.

How H7 · 1 can query them

Cloud Logging, project policyengine-observability, log policyengine-api-v1-modal:

jsonPayload."event.name"="operation.completed"
jsonPayload.outcome="interrupted"
jsonPayload."service.name"=~"^policyengine-(stage12|v2-household)-"

Each matching record's jsonPayload.attributes.database_table and jsonPayload.attributes.database_row_id name the row; read its final status from that table. jsonPayload.attributes.modal_call_id is the interrupted call, which report_runs, simulation_runs and household_jobs record as the call that claimed the row. A segment's call is recorded nowhere: its row is the simulation run that its simulation worker's call claimed.

Notes

  • The report-run parsing that budget_window_scheduling used moved into stage12_runtime/operation_records.py, unchanged, so the coordinator and the window scheduler share it.
  • The v1 code (src/modal/app.py and its modules) imports nothing added here. No image contents, secrets or environment variables change.
  • The new segment keyword is passed only when a simulation run is set. The segment function and the simulation workers deploy in the same app, but rolling the app back while new simulation workers are still running would make their spawns fail against the older segment signature.
  • Not fixed here: a few log calls outside the Entrypoint still pass plain extra= fields, which are not emitted: three in stage12_runtime/comparison.py, one in stage12_runtime/coordinated_simulations.py, one in the contract library's budget_window_state.py, and the shared library's errors.py used by the v1 gateway.

Tests

Test-first: c772f0c, deb60fa and b8eb69e add the failing tests. Only the new and affected tests were run, by node ID.

  • For every v2 function they check that the operation.completed record, after the package's filter, carries the three attributes, and that the metric attributes don't. They also check a call naming no row, a malformed ID, the US worker passing its run to its segments, and the segment runner's spawn keyword.
  • For the log fields: library tests log through the package's real handler; the Entrypoint tests check the four household job logs, the deferral log and the report and simulation logs, the INFO level, and the static rule with its proof fixtures.

Read-only review rounds: one on the v2 operations (no bugs); then three on the log fields. The first of those found that INFO records were never emitted, fixed test-first; the last found no bugs. A further round on the merge of main with #1090 and the ORM conversion also found no bugs. Quality findings from every round are fixed.

🤖 Generated with Claude Code

anth-volk and others added 9 commits November 17, 2026 13:00
… (failing)

Each v2 Modal function's operation record should carry `database_table`,
`database_row_id` and `modal_call_id` after the observability package's
attribute filter, and its metrics none of them. The US simulation worker
should pass its simulation run to the segments it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each v2 Modal function's operation now carries `modal_call_id` and, when
its call works on a detailed-save row, `database_table` (`household_jobs`,
`report_runs` or `simulation_runs`) and `database_row_id`. The US
simulation worker passes its simulation run to the segments it starts. The
keys avoid the words the observability package's filter drops, and the IDs
stay off metrics. The observability guide lists the new identifiers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… after review round 1

The US and UK simulation workers read their Modal call ID and simulation
run inside `try`, as the other v2 functions do. The observability guide
now says `database_table` is a telemetry-only name and that no row records
a segment worker's own Modal call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (failing)

The observability runtime's log handler emits only a record's
`policyengine_attributes`, and drops keys naming a household. A
`log_extra` helper in the shared observability library should pass each
field there; every Entrypoint log call with `extra=` should use it, with
no field named by a word the package drops; and the household job logs
should name their job as a `household_jobs` row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The observability runtime's log handler emits only a log record's
`policyengine_attributes`, so 31 of the Entrypoint's 33 log calls, which
passed their fields only as `extra=` keys, wrote records naming no job,
report, run or error type. A new `log_extra` helper in the shared
observability library puts the fields in both places, and every log call
now uses it. Rows are named with `database_table` and `database_row_id`
(`database_row_attributes`), since the runtime drops a key such as
`household_job_id`; the v2 Modal functions' operations take the same names
from the shared module. A budget window's started years are joined into
one string, since the runtime drops lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ble (failing)

After review round 2: the Entrypoint's INFO records should reach the
observability runtime's handler, which the root logger's WARNING level
stops today; a field named like a LogRecord attribute should be emitted
instead of making the log call raise; the economy, household report and
budget window logs should name their `reports` or `simulations` row; and
the static check gets proof fixtures for its rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er review round 2

The Entrypoint sets its package logger to INFO: the root logger's WARNING
level discarded its INFO records, such as a deferred budget-window year,
before the observability runtime's handler saw them. `log_extra` keeps a
field named like a LogRecord attribute only in `policyengine_attributes`,
since `logging` raises for it. One `DatabaseTable` type in the shared
module lists the tables telemetry names. Rule
`sim.observability.log-fields-through-log-extra` registers the static
check, and the observability guide's new section follows the identifier
table instead of splitting it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ck reads, after review round 3

`PACKAGE_LOGGER_NAME` comes from the module name, and the INFO-level test
imports it. The guide and the rule's statement say the check reads only
field names written as keywords in `log_extra(...)`, and that a field named
like a LogRecord attribute goes only to `policyengine_attributes`. The
budget-window log test checks the started years' value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk anth-volk changed the title Name the database row and Modal call in the v2 Modal functions' telemetry Name the database row and Modal call in v2 telemetry, and emit the Simulation Entrypoint's log fields Nov 17, 2026
anth-volk and others added 5 commits November 17, 2026 17:17
…rd IDs branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emetry record IDs branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d IDs branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g main

#1090 added `household_job_outcome_unknown`, which built
`policyengine_attributes` by hand; the rule this branch adds requires
`log_extra`. The emitted fields don't change. Its docstring now gives the
current reason it names no job. The text this branch added says API v1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With #1090 merged, 31 of the Entrypoint's 34 log calls that pass fields
were affected, and only the calls about a row name one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 18, 2026 22:34
@anth-volk
anth-volk requested a review from a team as a code owner November 18, 2026 22:34
@anth-volk
anth-volk merged commit 2b4a936 into main Nov 18, 2026
30 checks passed
@anth-volk
anth-volk deleted the v2plan/c1-34-v2-telemetry-record-ids branch November 18, 2026 22:34
anth-volk added a commit that referenced this pull request Nov 18, 2026
GitHub moved this pull request's base to main without a push, which starts
no PR checks run. This commit starts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 19, 2026
The three logs of a cliff calculation's call now reach Cloud Logging,
with the error's type and the cliff_impacts row, as #1092's rule
requires of every Simulation Entrypoint log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Name the database row and Modal call in the v2 Modal functions' telemetry

1 participant